Conventional digital paint systems use a graphic input device that tracks the position of a stylus and a processor that connects those points with lines in a display buffer. Some input devices can track the positions of multiple objects near their sensitive surface based on their images, as described by Hodges et. al in “ThinSight: Versatile Multi-touch Sensing for Thin Form-factor Displays” (Proceedings of the 20th annual ACM symposium on User interface software and technology, 2007, pp. 259-268) and Takahashi et al. in U.S. Pat. Nos. 8,085,351 and 8,189,128, the disclosures of which are incorporated by reference. That technology is embodied in a system manufactured by Samsung (Seoul, South Korea) called the SUR40, which includes software provided by Microsoft (Redmond, Wash.). In such systems, as the images of nearby objects are captured over time they are segmented and each segment is approximated by an ellipse whose position, size and orientation changes over time as the objects are moved. A paint system may then draw those ellipses that approximate the images of nearby objects such as fingers and brushes and connect the ellipses at successive positions of the object with lines, whereby the shape of the painted strokes more accurately reflects the changing shape of the objects.
In devices such as the SUR40 the painted image in progress may then also be displayed on the same surface, coincident with the input, and the device may be made very thin. However, because of feedback from the display to the sensitive portions of that surface, ghost images of the displayed output can appear in the input. This feedback causes nonuniformity in the responsiveness of such devices. The ghost images make them more sensitive to input where the displayed output is brighter and less sensitive where it is darker.
Other paint systems use the raw images of objects in optical contact with a drawing surface to provide their input, as described by Richard Greene in “The drawing prism: a versatile graphic input device” (ACM SIGGRAPH, Proceedings of the 12th annual conference on Computer graphics and interactive techniques, 1985, pp. 103-110) and U.S. Pat. Nos. 4,561,017, 5,181,108, and 8,022,927, the disclosures of which are incorporated by reference. When nothing is in contact with that surface, a background is imaged by total internal reflection (TIR). Only the portions of objects in optical contact with the surface, where there is no intervening layer of air to create TIR, are thereby imaged. Those images are then processed and accumulated in a display buffer to yield strokes that can capture the changing shape, texture, and shading of the areas of contact. While such systems provide a more expressive medium than those that only capture the positions or approximating ellipses of drawing tools, they also have some drawbacks. First, they require considerable depth below the drawing surface for their optical path, and that depth increases with the size of the active drawing surface. Second, they require the drawing tools to be wet, in order to eliminate the layer of air that provides TIR. Finally, they do not provide a means for locating the input coincident with the display of the image as it is being painted. Therefore users are forced to look away from the drawing tool in order to see the display of the painting itself. Also, when they first touch the tool to the drawing surface, they cannot tell exactly where in the painting that will leave its mark, until after they see where that mark was made.
Devices such as the SUR40 detect portions of objects that may be inches from their sensitive surface. They do not just detect the portions of objects in optical contact with that surface, i.e. where there is no air between the object and the surface.